[CAI-749] Local parser by anemone008 · Pull Request #2007 · pagopa/developer-portal

anemone008 · 2026-02-09T16:44:33Z

List of Changes

Add script for parsing to parser app. Parsed content is saved locally. Urls are sanitized for filesystems and used as file names.

Motivation and Context

How Has This Been Tested?

Tested for errors associated to non-existent or unreachable urls. Reproducible through npm test as described in the README.md

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

…r filesystem use.

changeset-bot · 2026-02-09T16:44:38Z

🦋 Changeset detected

Latest commit: 2a1b9a0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
parser	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copilot

Pull request overview

Adds a local “parser” CLI to crawl a site, extract page metadata, and persist results to disk with URL/filename sanitization.

Changes:

Introduces a Puppeteer-based crawler/metadata extractor and local JSON output.
Adds URL normalization + filesystem-safe filename sanitization utilities.
Adds build/test tooling (TypeScript + Jest) and an error-handling integration test.

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
apps/parser/src/parser.ts	Adds the CLI entrypoint: reachability check, crawl orchestration, page parsing, persistence.
apps/parser/src/modules/crawler.ts	Implements recursive crawl + scope filtering + link discovery.
apps/parser/src/modules/domActions.ts	Expands interactive UI sections before scraping text/links.
apps/parser/src/modules/config.ts	Resolves env-based configuration and output directory derivation.
apps/parser/src/modules/output.ts	Creates output directory and writes JSON snapshots.
apps/parser/src/modules/errors.ts	Centralizes fatal error handling/exit code.
apps/parser/src/modules/types.ts	Adds typed metadata/node structures for crawl results.
apps/parser/src/utils/url.ts	Adds URL normalization helpers and remote URL detection.
apps/parser/src/utils/sanitizeFilename.ts	Adds filesystem-safe filename sanitization.
apps/parser/tests/parser.error-handling.test.ts	Adds integration test for unreachable/nonexistent URL behavior.
apps/parser/package.json	Adds build/parse/test scripts and required dependencies.
apps/parser/jest.config.ts	Configures Jest + ts-jest for the parser app.
apps/parser/tsconfig.json	Adds parser app TS config for dev/test typechecking.
apps/parser/tsconfig.build.json	Adds build TS config emitting to dist/.
apps/parser/README.md	Documents CLI usage, env vars, and tests.
.changeset/wide-hairs-fail.md	Changeset entry for the new parser feature.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/parser/src/parser.ts

apps/parser/src/modules/domActions.ts

apps/parser/src/modules/output.ts

apps/parser/src/modules/crawler.ts

apps/parser/src/parser.ts

apps/parser/tests/parser.error-handling.test.ts

apps/parser/README.md

apps/parser/src/utils/sanitizeFilename.ts

apps/parser/src/modules/config.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ng of the same document

…a/developer-portal into CAI-749-parser-url-crawler

…es uniqueness

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…le navigation_timeout parameter. Remove unused isRemoteUrl

…awler

apps/parser/README.md

apps/parser/src/modules/config.ts

apps/parser/src/modules/crawler.ts

apps/parser/src/modules/dom-actions.ts

apps/parser/src/utils/sanitizeFilename.ts

apps/parser/src/utils/url.ts

apps/parser/src/parser.ts

…nd version bump

Co-authored-by: MarBert <41899883+MarBert@users.noreply.github.com>

…awler

… code and add new helper for dates

MarBert · 2026-02-12T13:56:53Z

apps/parser/src/helpers/url-handling.ts

+  if (!input) {
+    return DEFAULT_REPLACEMENT;


i suggest to add a warning log so that we can understand if something went wrong and the DEFAULT_REPLACEMENT is used

apps/parser/src/helpers/url-handling.ts

MarBert · 2026-02-12T13:58:48Z

apps/parser/src/helpers/url-handling.ts

+  return candidate;
+}
+
+export const UrlWithoutAnchors = (rawUrl: string): string => {


i suggest a name like RemoveAnchorsFromUrl, it's easier to understant what this does

MarBert · 2026-02-12T14:10:14Z

apps/parser/src/modules/crawler.ts

+  try {
+    page = await browser.newPage();
+    await page.goto(node.url, {
+      waitUntil: "networkidle2",


can we use a constant for this?

MarBert · 2026-02-12T14:12:45Z

apps/parser/src/modules/crawler.ts

+    url.hash = "";
+    return UrlWithoutAnchors(url.toString());
+  } catch (_error) {
+    return rawUrl;


i suggest to add a warning here

MarBert · 2026-02-12T14:27:01Z

apps/parser/src/parser.ts

+  }
+}
+
+async function persistSnapshot(snapshot: ParsedMetadata, FILENAME_LENGTH_THRESHOLD: number): Promise<void> {


isn't FILENAME_LENGTH_THRESHOLD a const reachable inside this function ? Why do you need to pass it as a parameter?

MarBert · 2026-02-12T14:33:26Z

apps/parser/src/parser.ts

+      0,
+      env.maxDepth,
+      parsedPages,
+      parsePageFn,


why do you have to pass the function here? Can't we export the function and call it inside parsePages?

apps/parser/src/modules/config.ts

MarBert

Aggiungere un log con testo "Completed parsing of page [...]. Found [...] links. [nr liks parsed/nr links to parse]" per monitorare l'avanzamento del programma;
Rimuovere il trattino finale nei nomi dei file .json;
Gestire il problema del dominio con uno split al 1° slash dopo https:// e verifica che sia uguale al baseUrl oppure a {variante_valida}.dominio - aggiungere un TODO per soluzione più generale;
Diminuire FILENAME_LENGTH_THRESHOLD a 250, in modo che ci sia spazio per l'estensione nel filename;
Fare in modo che il nome della cartella dove sono salvati i json sia del tipo www.uqido.com, senza http... iniziale;
Chiamare _.json il file relativo alla homepage - poi chiedere conferma a Ciri se servono nomi specifici;
Rinominare la funzione parsePages in exploreAndParsePages;
Rinominare il file parser.ts in main.ts e spostare tutte le funzioni, a eccezione del lancio della ricorsione, negli helpers;
Rinominare crawler.ts in parser.ts e spostarci la funzione parsePageFn, rinominata in generatePageParsedMetadata;
Verificare quali sono i parametri che non è necessario passare alla ricorsione ed esportarli come un nuovo oggetto recursionMetadata;

- Update sanitizeUrlAsFilename to support length threshold and hash suffix for long URLs. - Modify resolveEnv to parse validDomainVariants from environment variables. - Refactor parsePages to utilize validDomainVariants for scope checking. - Update EnvConfig type to include validDomainVariants. - Improve tests for sanitizeUrlAsFilename to cover new functionality.

…me function so that for root it returns the hostname

… UrlWithoutAnchors to RemoveAnchorsFromUrl

…,remove constants from recursion parameters and enhance helpers

github-actions · 2026-02-13T11:28:11Z

Branch is not up to date with base branch

@anemone008 it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

…awler

…mited depth, implement base scope handling, and enhance URL sanitization functions for Directory names

github-actions · 2026-02-13T15:49:05Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-749

github-actions · 2026-02-13T15:49:09Z

This PR exceeds the recommended size of 800 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

anemone008 added 2 commits February 9, 2026 17:24

Add script to perform parsing. Store output locally. Sanitize urls fo…

dbcf818

…r filesystem use.

Add changeset

4354cd5

anemone008 requested review from a team, MarBert, MarcoPonchia, marcobottaro and tommaso1 as code owners February 9, 2026 16:44

github-actions bot added the changeset label Feb 9, 2026

anemone008 self-assigned this Feb 9, 2026

anemone008 requested a review from Copilot February 9, 2026 16:45

Copilot AI reviewed Feb 9, 2026

View reviewed changes

anemone008 marked this pull request as draft February 9, 2026 16:59

anemone008 added 2 commits February 9, 2026 18:08

Fix fetch. Set headless true

a9540a3

Replace timeout with AbortController + setTimeout

de11fe0

batdevis reviewed Feb 10, 2026

View reviewed changes

apps/parser/src/modules/config.ts Outdated Show resolved Hide resolved

anemone008 and others added 7 commits February 10, 2026 10:12

Add filter on ariaExpanded to avoid collapse of already expanded content

3daa8c9

Update apps/parser/src/modules/output.ts

a1ea8f0

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Drop the hash from the visit key to avoid redundant navigation/scrapi…

4018873

…ng of the same document

Merge branch 'CAI-749-parser-url-crawler' of https://github.com/pagop…

3feea1b

…a/developer-portal into CAI-749-parser-url-crawler

Update README.md

26236f0

Add test on filename sanitization

9afad68

Add short hash when filename is longer than 255 chars for URL filenam…

d0b0cc6

…es uniqueness

github-actions bot added the size/large label Feb 10, 2026

anemone008 and others added 2 commits February 10, 2026 12:04

Update apps/parser/tests/parser.error-handling.test.ts

31c4bad

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Rename VECTOR_INDEX_NAME to PARSER_VECTOR_INDEX_NAME

fca22ae

anemone008 marked this pull request as ready for review February 10, 2026 13:56

anemone008 and others added 3 commits February 10, 2026 17:42

Refactor deriveSubPath to use base URL for path derivation

748684e

Rename vector_index variable, remove alternative output dir, set sing…

b403294

…le navigation_timeout parameter. Remove unused isRemoteUrl

Merge remote-tracking branch 'origin/main' into CAI-749-parser-url-cr…

ccdc78a

…awler

MarBert requested changes Feb 11, 2026

View reviewed changes

anemone008 and others added 12 commits February 11, 2026 16:45

Update README and config for improved environment variable handling a…

eff7c31

…nd version bump

Update apps/parser/README.md

e53dfa7

Co-authored-by: MarBert <41899883+MarBert@users.noreply.github.com>

Refactor URL handling and filename sanitization into helpers

f1ac135

Move assertReachable function to module network.ts

1be8fc9

Rename buildDefaultOutputDirectory to generateOutputDirectoryPath

d2a476a

Merge remote-tracking branch 'origin/main' into CAI-749-parser-url-cr…

bb35a84

…awler

Add error logging for failed anchor href parsing in parsePages function

f94b3a6

Add warning logs for iframe src parsing and anchor extraction failures

58ec25e

Improve error handling in UrlWithoutAnchors

e291067

Move FILENAME_LENGTH_THRESHOLD constant at the start of the file

9de4eb4

Update README for type-check and compile instructions

462c5e6

Refactor code for consistency and readability; update imports, format…

c8b1cab

… code and add new helper for dates

MarBert requested changes Feb 12, 2026

View reviewed changes

apps/parser/src/modules/config.ts Show resolved Hide resolved

MarBert requested changes Feb 12, 2026

View reviewed changes

anemone008 added 7 commits February 13, 2026 09:58

Add progress log in recursive parsePages function

0c19cbb

Add TODO in isWithinScope

985363d

Rename parsePages to exploreAndParsePages. Expand sanitizeUrlAsFilena…

eb5e964

…me function so that for root it returns the hostname

Add warnings for missing or invalid URL replacements; rename function…

416ef96

… UrlWithoutAnchors to RemoveAnchorsFromUrl

Refactor parser module: clean main function, rename crawler to parser…

5d24926

…,remove constants from recursion parameters and enhance helpers

Formatting

cfd4ffe

anemone008 added 3 commits February 13, 2026 12:28

Merge remote-tracking branch 'origin/main' into CAI-749-parser-url-cr…

923aa16

…awler

Save root page metadata as index.json

755fb6b

Refactor parser configuration: update maxDepth to allow null for unli…

2a1b9a0

…mited depth, implement base scope handling, and enhance URL sanitization functions for Directory names

MarBert self-requested a review February 13, 2026 16:15

Conversation

anemone008 commented Feb 9, 2026

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MarBert Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MarBert left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 13, 2026

Branch is not up to date with base branch

Uh oh!

github-actions bot commented Feb 13, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

github-actions bot commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

changeset-bot bot commented Feb 9, 2026 •

edited

Loading

github-actions bot commented Feb 13, 2026 •

edited by atlassian bot

Loading